Gated XNOR Networks: Deep Neural Networks with Ternary Weights and Activations under a Unified Discretization Framework

摘要

There is a pressing need to build an architecture that could subsume thesenetworks undera unified framework that achieves both higher performance andless overhead. To this end, two fundamental issues are yet to be addressed. Thefirst one is how to implement the back propagation when neuronal activationsare discrete. The second one is how to remove the full-precision hidden weightsin the training phase to break the bottlenecks of memory/computationconsumption. To address the first issue, we present a multistep neuronalactivation discretization method and a derivative approximation technique thatenable the implementing the back propagation algorithm on discrete DNNs. Whilefor the second issue, we propose a discrete state transition (DST) methodologyto constrain the weights in a discrete space without saving the hidden weights.In this way, we build a unified framework that subsumes the binary or ternarynetworks as its special cases.More particularly, we find that when both theweights and activations become ternary values, the DNNs can be reduced to gatedXNOR networks (or sparse binary networks) since only the event of non-zeroweight and non-zero activation enables the control gate to start the XNOR logicoperations in the original binary networks. This promises the event-drivenhardware design for efficient mobile intelligence. We achieve advancedperformance compared with state-of-the-art algorithms. Furthermore,thecomputational sparsity and the number of states in the discrete space can beflexibly modified to make it suitable for various hardware platforms.